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BACKGROUND OF THE INVENTION 



FIELD OF THE INVENTION 



10 



The present invention relates to the field of image recognition, and in particular to 



a method and apparatus for accelerated handwritten symbol recognition in a pen based 
tablet computer. 

2. BACKGROUND ART 

15 

In some computer systems, handwritten symbols are input to the system. These 
symbols are translated by the computer system to machine readable characters. This 
translation is typically computation intensive. In some computer systems, battery 
operated portable devices for example, the general purpose central processing unit (CPU) 

20 used for the translation is inefficient in its power consumption during the translation 

operation. Thus, the battery is drained more rapidly. Additionally, some battery operated 
systems are limited in computational power. When a real-time translation requirement is 
placed on symbol translation, the limited computational power results in a limited degree 
of accuracy in the translation process. These problems can be better understood with a 

25 review of handwritten data entry. 
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Handwritten Data Entry 



A typical computer system consists of a central processing unit (CPU), main 



5 memory such as random access memory (RAM), a data entry device, including a 
positioning device, a mass storage device such as one or more disk drives, a display 
and/or a printer. In the prior art, the data entry device often consists of a keyboard, on 
which a user enters data by typing. The positioning device of a prior art computer system 
may consist of a "mouse" or other cursor positioning device. 



Computer systems also exist that are directed to handwritten data entry rather than 
keyboard data entry. These systems are often characterized by the use of a pen, stylus, or 
other writing device, to enter handwritten data directly on the display of the computer 
system. Alternatively, these systems may provide for a user to enter data on a digitizing 

15 tablet or other input device, with the image of the written input displayed on a separate 
computer display output device. The writing device for entering handwritten or freestyle 
stroke input information is not limited to a pen or stylus, but may be any input device 
such as a mouse, trackball, pointer, or even a person's fingers. Such systems are not 
necessarily limited to receiving data generated by human users. For example, machine 

20 generated data may also be inputted and accepted to such systems. 

One class of this handwriting entry computer system that receives handwritten 
data input is referred to as a "pen based" computer system, hi a pen based computer 
system, a writer can input information on a display by "writing" directly on the display. A 
25 writing device, such as a pen or stylus, is used to enter information on the display, hi a 
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typical pen-based computer system, a user touches the stylus to the display and writes as 
the user would on a piece of paper, by making a series of pen strokes to form letters and 
words. A line appears on the display that follows the path of travel of the pen point, so 
that the pen strokes appear on the display as ink would appear on a handwritten page. 
5 Thus, the user can enter information into the computer by writing on the display. Pen 
based computers typically have a display surface that serves as both an input receiving 
device and as an output display device. 

Handwritten Data Translation 

10 

One characteristic of handwriting entry computer systems is the ability to translate 
original handwritten symbols into machine readable words or characters for display. This 
translation is accomplished via a "character recognition" algorithm. The handwritten 
symbols are translated into, for example, ASCII characters. After the translation, the 
1 5 appearance of the displayed characters is as if they had been typed in via a keyboard. 

To translate a handwritten character into a machine readable character, the 
handwritten character is compared to a library of characters to determine if there is a 
match. A description, or "template" for each character is defined and stored in memory. 

20 Handwritten characters are compared to the stored templates. Match coefficients, 

reflecting how closely a handwritten character matches the template of a stored character, 
are calculated for each template character. The template character with the highest match 
coefficient is identified. The character represented by this template provides the "best fit" 
for the handwritten character. If the match coefficient for the "best fit" character exceeds 

25 a predetermined minimum threshold, the "best fit" character is adopted. If the match 
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coefficient for the "best fit" character is less than the minimum threshold value, no 
translation is done. If the handwritten character cannot be translated, the character must 
be re-entered. 

5 A disadvantage of current character recognition algorithms is limited accuracy. 

Often, handwritten characters are not translated at all or are mistranslated as an ASCII 
character other than the handwritten character. The mistranslated character must then be 
rewritten by the user, sometimes repeatedly, until a correct translation is made. 

10 Handwriting Recognition in Portable Systems 

ffi 

5 A portable pen-based computer systems is constrained by the amount of power 

U~l stored in its battery. Typically, portable pen-based computer systems, which require 

^ handwriting recognition (HWR), rely on grid based single character recognition, which 

o 

W 15 forces users to print characters in stylized formats. This approach is not suitable for 
bj entering large text segments. A better approach for entering large text segments is to 

M= enable users to vmte naturally on the screen in their ovra personal, unconstructed style 

using HWR algorithms. However, HWR algorithms require a large amount of 
computation to translate handwritten symbols into machine readable characters. Typical 
20 portable pen-based computer systems lack the computational power necessary to 
satisfactorily perform translations. 

Typical portable pen-based computer systems use a general purpose CPU for 
HWR calculations. Typically, a general purpose CPU is inefficient in power 
25 consumption during HWR calculations. The general purpose CPU is designed to perform 

5 
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more than HWR calculations, so some functions of the CPU are powered, but not used 
for the HWR calculation. Additionally, a general purpose CPU is inefficient in speed 
during HWR calculations. The general purpose CPU must be able to perform certain 
operating system tasks while completing HWR calculations. Thus, the speed with which 
5 HWR calculations are completed is diminished. As a result, fewer HWR calculations 
may be completed in a limited amount of time. Thus, if the time for HWR is limited, the 
accuracy of the translation is also limited. 

Single Symbol Translation 

^ Typically, portable pen-based computer systems translate one character at a time. 

4= 

ffl However, such a scheme is difficult when a user has poor handwriting. For example, 

Ul Figure 1 illustrates a word where single symbol translation is difficult. A single symbol 

translation system has difficulty translating the letters "m" (100), "i" (1 10), and "n" (120) 
m 15 in the word "jumping" (130) in Figure 1. However, the word is legible to the user. 
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SUMMARY OF THE INVENTION 

The present invention provides a method and apparatus for accelerated 
handwritten symbol recognition in a pen based tablet computer. In one embodiment, 
5 handwritten symbols are translated into machine readable characters using hidden 

Markov models. In one embodiment, handwritten symbols are translated into machine 
readable characters using special purpose hardware. In one embodiment, the special 
purpose hardware is a recognition processing unit (RPU) which performs feature 
extraction and recognition. A user inputs the handwritten symbols and software 
10 recognition engine preprocesses the input to a reduced form. In one embodiment, the 
preprocessor is fully information preserving. 

The data from the preprocessor is sent to the RPU which performs feature 
extraction and recognition. In one embodiment, the RPU has memory and the RPU 
1 5 operates on data in its memory. In one embodiment, the RPU uses a hidden Markov 

model (HMM) as a finite state machine that assigns probabilities to a symbol state based 
on the preprocessed data from the handvmtten symbol. In another embodiment, the RPU 
recognizes collections of symbols, termed "wordlets," in addition to individual symbols. 

20 In one embodiment, the software recognition engine uses the data from the RPU 

in a postprocessor. The postprocessor computes a stream of symbol observation events 
from data produced by the RPU and writer confirmation data. In one embodiment, the 
postprocessor also uses information about context, spelling, grammar, past word usage 
and user information to improve the accuracy of the symbols produced. 

7 
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BRffiF DESCRIPTION OF THE DRAWINGS 

These and other features, aspects and advantages of the present invention will 
become better understood with regard to the following description, appended claims and 
5 accompanying drawings where: 

Figure 1 is a block diagram of a word where single symbol translation is difficult. 

Figure 2 is a block diagram of a pen-based (tablet) computer handwriting symbol 
10 recognition system in accordance with one embodiment of the present invention. 

Figure 3 is a flow diagram of the process of translating handwritten symbols in 
accordance with one embodiment of the present invention. 

15 Figure 4 is a flow diagram of the process of translating handwritten symbols 

wherein the RPU calculates forward and backward probabilities in accordance with one 
embodiment of the present invention. 

Figure 5 is a flow diagram of the process of translating handwritten symbols 
20 wherein the RPU calculates forward probabilities in accordance with one embodiment of 
the present invention. 

Figure 6 is a block diagram of an RPU in accordance with one embodiment of the 
present invention. 

25 
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Figure 7 is a flow diagram of the process of handwritten symbol translation using 
context information in accordance with one embodiment of the present invention. 

Figure 8 is a flow diagram of the process of handwritten symbol translation using 
5 spelling information in accordance with one embodiment of the present invention. 

Figure 9 is a flow diagram of the process of handwritten symbol translation using 
grammar information in accordance with one embodiment of the present invention. 

10 Figure 10 is a flow diagram of the process of handwritten symbol translation using 

past word usage information in accordance with one embodiment of the present 
invention. 



Figure 1 1 is a flow diagram of the process of handwritten symbol translation using 
15 user information in accordance with one embodiment of the present invention. 



9 



LA49978vl 



DETAILED DESCRIPTION OF THE INVENTION 

The invention is a method and apparatus for accelerated handwritten symbol 
recognition in a pen based tablet computer. In the following description, numerous 
5 specific details are set forth to provide a more thorough description of embodiments of 
the invention. It is apparent, however, to one skilled in the art, that the invention may be 
practiced without these specific details. In other instances, well known features have not 
been described in detail so as not to obscure the invention. 

10 Handwriting Recognition Calculations 



In one embodiment, HMM calculations are used to determine the probability of a 
symbol appearing in a sequence of symbol observations. A HMM with N states, M 
observation symbols, the state alphabet Vs = {Si, S2, . . Sn} and the emission alphabet 
15 Ve = {vi, V2, . . vm} is defined by the triplet X = (A, B, 7t). A is a state transition matrix 
defined as ay = P(qt+i = Sj | qt = Si) for 1 < z < and 1 < j <N , which means the 

probability that the state at time t+1 is state j given that the state at time t is state i. B is 
the observation probability matrix defined as bj(k) = P(vk | qt = Sj) for 1 < 7 < and 

l<k <M , which means the probability of the observation being Vk given that the state at 
20 time t is state j. p is an initial state distribution defined as = P(qi = Si), which means 
the probability of the state at time 1 is state i. 



If we have an observation sequence O = (01O2. . .Ot), the RPU calculates the 
probability of this sequence given the model X. This value is calculated by determining a 
25 series of values termed forward variables defined as ai(t) = P(oi02. . .Ot, qt = Si | X), which 
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• I 



means the probability of the observation sequence from 1 to t and the state at time t being 
state i given X. These values are calculated by initializing the variable as ai(l) = 7ribi(oi) 
for l<i<N , Further values are calculated using aj(t+l) = bi(ot+i) ^ ai(t)aij for 

1 < / < 7 - 1 and 1 < j < N , The probability of the sequence given X is defined by P(P | 

5 X)=f^ ai(T). 
/"=i 



Similarly, a backv^ard variable is defined as Pi(t) = P(ot+iOt+2. . .oj | qt = Si, X), 
which means the probability of the observation sequence from t+1 to T given state at time 
a t being state i given X, The backward variables are initialized as pi(T) = 1 for 1 < / < TV . 
S 10 Further values are calculated using pj(t) = ^ aijbj(ot+i)pi(t+l) for 1 < / < T - 1 and 



^ The calculations to compute forward and backward variables are performed in the 

S=; RPU. Thus, probabilities can be calculated for each new symbol to determine which 

% 15 symbols the new symbol is most likely to be. In one embodiment, the HMM calculations 

^ are performed on a general purpose computational unit. 



Pre-processed symbol observations are the input to the HMMs. In one 
embodiment, the symbol observation alphabet (the emission alphabet) is comprised of 
20 angles of equalized segments. In other embodiment, more complex symbol observation 
alphabets are used. In one embodiment, at least one HMM is created for each symbol in 
the output alphabet. The probability of each symbol given the observation sequence is 
calculated by each symbol's HMM. In one embodiment, a post-processing unit uses the 
information from the HMMs to determine the appropriate symbol. 
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Training A, B and n 



It is desirable to select the parameters A, B and nofX that maximize the 
5 probability of a sequence in the training set given X. One algorithm used to determine the 
parameters is the Baum- Welch method. The Baum- Welch method guarantees a 
monotonically increasing probability and converges quickly. 



First, a joint event variable is defined as eij(t) = P(qt = Si, qt+i =^ Sj | O, X\ which 
% 10 means that the probability of the state at time t being state i and the state at time t+1 being 
^ state j given sequence O and X. From the definitions of forward and backward variables, 

5 this becomes 8ij(t) = (ai(t)aijbj(oHi)Pj(t+l)) / P(0 | X), 



Additionally, a state variable is defined as Yi(t) = P(qt = Si | O, X\ which means the 
Uj 15 probability of the state at time t being state i given sequence O and X,. From the 
yj definitions of forward and backward variables, this becomes Yi(t) = (ai(t)pj(t)) / P(0 | X), 

r-1 

A new X, X\ is calculated as follows. A new a, a', is calculated as a'ij = ^ Sij(t) 

t=\ 

r-1 r r 

/ ^ Yi(t). A new b, b', is calculated as b'j = ^ yj(t) / ^ Yj(t). A new tt, n\ is 

^=1 /=l,0,=Vt f=l 

20 calculated as Ti'i = Yi(l). 



A variation of the Baum- Welch method, termed the "Levingson method," 

calculates X' as follows when K observation sequences are used to adjust the parameters. 

K r-1 K r-1 

A new a, a', is calculated as a'ij = X! Z ^u^^'*^^) ^ Z Z Yi^^^O- A new b, b', is 

k=\ t=\ k=\ r=l 
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K T K T 

calculated as b'j^*"^ = ^ ^ Yj^^\t) / J] ^ yj^^^t). A new tt, %\ is calculated as Ti'i 

k-\ /=l,o,=vt A=l /=1 

= 1/K * Yi^^^i). 

Special Purpose Hardware for Recognition Processing 

5 

In one embodiment, handwritten symbols are translated into machine readable 
characters using special purpose hardware, hi one embodiment, the special purpose 
_ hardware is a recognition processing unit (RPU) which performs feature extraction and 
recognition. Li another embodiment, a user inputs the handwritten symbols and software 
10 recognition engine preprocesses the input to a reduced form, hi one embodiment, the 
~Q preprocessor is fully information preserving. 

Q Figure 2 illustrates a pen-based (tablet) computer handwriting symbol recognition 

Li \ 

n system in accordance with one embodiment of the present invention. A writer (200) 
S 15 enters handwritten symbols into the tablet computer (210). The handwritten symbols 

operated upon by a preprocessor (220) running on the main processing unit (MPU). The 
data from the preprocessor is sent to an RPU (230). The RPU is implemented as special- 
purpose hardware. Li one embodiment, the RPU is a circuit configured to perform hidden 
Markov model (HMM) computations. The data from the RPU is used by a postprocessor 
20 (240) running on the MPU to produce an unconfirmed symbol observation (250). 

The imconfirmed symbol observation is presented to the vmter. The vmter can 
confirm a symbol, reject a symbol or make no determination. The postprocessor uses 
confirmed symbol observations (260), rejected symbol observations and unconfirmed 
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symbol observations to adjust how it makes symbol observations. The preprocessor also 
uses confirmed symbol observations and unconfirmed symbol observations to adjust how 
it preprocesses the handwritten symbols. Additionally, training data (270) is used by the 
preprocessor, the RPU, and the postprocessor to adjust their calculation to achieve more 
5 accurate symbol translations. 

The special purpose hardware of the RPU enables the system to calculate more 
handwriting recognition calculations in the same amount of time when compared to a 
system where handwriting recognition calculations are performed by a general purpose 

10 processor (the MPU). In one embodiment, the RPU uses parallel processing to make 
multiple handwriting recognition calculations each clock cycle. Typically, a general 
purpose processor requires multiple clock cycles to perform one handwriting recognition 
calculation. Li one embodiment, the RPU performs eight handwriting recognition 
calculations in parallel for each clock cycle. Since the RPU only performs handwriting 

15 recognition calculations, no power is wasted during the calculation. Thus, the same 

amount of handwriting recognition calculations will require less power when computed 
by an RPU than when computed by a general purpose processor. 

Memory on the RPU 

20 

In one embodiment, the data from the preprocessor is sent to the RPU which 
performs feature extraction and recognition. In another embodiment, the RPU has 
' memory and the RPU operates on data in its memory. Figure 3 illustrates the process of 
translating handwritten symbols in accordance with one embodiment of the present 
25 invention. At step 300, a user enters handwritten symbols into the system. At step 310, 
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the handwritten symbol data is stored in memory accessible by the MPU. At step 320, a 
preprocessor running on the MPU operates on the handwritten symbols. At step 330, the 
data from the preprocessor is transferred to the memory of an RPU. At step 340, the RPU 
operates on the data in its memory. At step 350, the data from the RPU is transferred to 
5 the memory accessible by the MPU. At step 360, a postprocessor running on the MPU 
generates a symbol observation. 

Figure 4 illustrates the process of translating handwritten symbols wherein the 
RPU calculates forward and backward probabilities in accordance with one embodiment 

10 of the present invention. At step 400, a user enters handwritten symbols into the system. 
At step 410, the handwritten symbol data is stored in memory accessible by the MPU. At 
step 420, a preprocessor running on the MPU operates on the handwritten symbols. At 
step 430, the data from the preprocessor is transferred to the memory of an RPU. At step 
440, the RPU calculates forward and backward probabilities. At step 450, the data from 

1 5 the RPU is transferred to the memory accessible by the MPU. At step 460, a 
postprocessor running on the MPU generates a symbol observation. v 

Figure 5 illustrates the process of translating handwritten symbols wherein the 
RPU calculates forward probabilities in accordance with one embodiment of the present 

20 invention. At step 500, a user enters handwritten symbols into the system. At step 505, 
the handwritten symbol data is stored in memory accessible by the MPU. At step 5 10, a 
preprocessor running on the MPU operates on the handwritten symbols. At step 515, the 
data from the preprocessor is transferred to the memory of an RPU. At step 520, the RPU 
calculates initial forward probabiUties, ai(l). At step 525, the RPU calculates successive 

25 forward probabilities (ai(2), ai(3), . . ., ai(t)) to determine the probabilities of the observed 

15 
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sequences. At step 530, the data from the RPU is transferred to the memory accessible by 
the MPU. At step 535, a postprocessor running on the MPU generates a symbol 
observation. 

5 Figure 6 illustrates an RPU in accordance with one embodiment of the present 

invention. The RPU 600 has a memory unit 610 which stores aij 620, bi(ot+i) 630 and 
values of ai(t) 640 as they are calculated. Initially, the values of ai(t) are set to their 
initial values. Additionally, the RPU has a HMM calculation unit 650. The HMM has N 
ai(t) units 660. Each ai(t) unit is multiplied by the appropriate a^ value in an aij unit 670. 
10 All of the products are summed in a summation unit 680 and the sum is multiplied by the 
appropriate bi(ot+i) value stored in a bi(ot+i) unit 690. The resulting product is stored in 

N 

the memory unit. Thus, the HMM calculates aj(t+l) = bi(ot+i) ^ ai(t)aij for 1 < / < T - 1 
and 1 < 7 < . 

15 In some embodiments, the RPU has multiple HMM calculation units to enable 

multiple HMM calculations to take place in parallel. In one embodiment, the RPU has N 
HMM calculation units. Thus, values of aj(t-i-l) are calculated in parallel for all values of 

j- 

20 Symbols and Wordlets 

In one embodiment, the RPU uses a hidden Markov model (HMM) as a finite 
state machine that assigns probabilities to a symbol based on the preprocessed data from 
the handwritten symbol. For example, a handwritten symbol may have a one in three 

25 probability of being an "e" and a one in four probability of being an "i." In another 

-\ 
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embodiment, the RPU recognizes collections of symbols, temied "wordlets," in addition 
to individual symbols. 

For example, the RPU may recognize "tion" or "ing" as one symbol. The output 
5 alphabet contains "tion" and "ing" in addition to "t", "i", "o", "n" and "g". The ability to 
recognize a handwritten symbol as the wordlet "ing" improves the accuracy of translation, 
For example, in Figure 1, the "i," the "n" and the "g" are all difficuh to recognize 
individually. However, the "ing," together, is easier to recognize. Where one letter ends 
and the next begins becomes less important to the determination when the entire wordlet 
1 0 can be recognized by the RPU. 



Probabilistic Context Free Grammar 



Li one embodiment, probabilistic context free grammar information is used to 
15 improve the accuracy of symbol translation. A probabilistic context free grammar is 

defined as G = (Vn, Vt, P, S). Vn is a nonterminal feature alphabet defined as Vn = {Fi, 
F2, . . Fn}. Vt is a terminal feature alphabet defined as Vj = {wi, W2, . . wm}. All of 
the production rules of the grammar are of the form Fi FjFk or Fi -> w^ where Fj e Vn 
are nonterminal features and Wk € Vt are terminal features. Fi is set equal to the entire 
20 string of terminals. 



These production rules are defined by tensors A and B as P = (A, B). For the 
nonterminal features, a probability tensor A of rank 3 is defined as aijk = P(Fi FjFk) for 
l<i<N , 1 < y < and l<k<N, For the terminal features, a production probability 
25 matrix B is defined as bj(k) = P(Fj -> Wk) for 1 < 7 < and 1 < A: < M . 
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In one embodiment, the probability of a string of terminals of length T, W == 
w . .w\ where € Vj is determined given a probabilistic context free grammar 
defined as P(W | G). In one embodiment, the probability of a sub-sequence, W^'^ = 
5 w*'. . .w*', termed an "inside probability" is calculated. The inside probability is initiaUzed 
as Pi(t, t) = bi(w^) for l<i<N and 1 < / < T . Successive inside probabilities are 

N N 9-1 

determined by calculating pi(p, q) = X S ^'J^ S Pi(P'OPk(t+l ,q) for I <i<N . At 

7=1 t=p 

termination, P(W | G) = P i ( 1 ,T). 
10 Similarly, the "outside probability" is the probability that the sub-sequence, W^''* = 

12 T 

w^. . .w^, was generated by the nonterminal Fi in the sequence W = w w . . .w . The 
outside probability is initiaUzed as ai(l, T) = 5ii for \ <i<N , Sh = 1 for i = 1 and b\\ = 0 
for all other values of i. Successive outside probabilities are determined by calculating 

N N T N N p-\ 

aj(p,q)=X E ^^Y. ai(P,t)Pk(q+l,t)+ ak(t,q)pi(t,p-l) for 

/=1 k=\,j^k t^q^l i=\ *=1 ^=1 

N 

15 1 < 7 < . At termination, P(W | G) = ^ bj(wVj(t,t) for 1 < ^ < T . 
Training for Probabihstic Context Free Grammar 



A joint feature probability is defined as £»ijk(p, q) = 

20 — . A parent feature probability is defined as yi(p, q) = 

^i(p^q)fiiiP^q) • -j^^ feature probability and parent feature probability are used to 
MIT) 

calculate new nonterminal probabilities and new terminal probabilities. The new 
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r-i T 

nonterminal probabilities are calculated as aijk - -^—^ for l< j < N and 

p=\ g=p 

T 

\<k<M , The new terminal probabilities are calculated as b > {k) = ^"^'y,""^* for 

p^\ g=p 

l<j<N mdl<k<M , 



5 In one embodiment, the inside probability is used to determine a probability of a 

string of observed terminals given a probabilistic context free grammar. In one 
embodiment, the tensors of the probability are trained on a sample language by 
calculating nev^ terminal and nonterminal probabilities using the above equations. In one 
embodiment, the inside probability is calculated using general purpose hardv^are. In 
10 another embodiment, the inside probability is calculated using special purpose hardware. 



Context Consideration to Improve Accuracy 



In one embodiment, the software recognition engine uses the data from the RPU 
15 in a postprocessor. The postprocessor computes a stream of symbol observation events 
from data produced by the RPU and vmter confirmation data. In one embodiment, the 
postprocessor also uses information about context, spelling, grammar, past word usage 
and user information to improve the accuracy of the symbols produced. 



20 Figure 7 illustrates the process of handvmtten symbol translation using context 

information in accordance with one embodiment of the present invention. At step 700, a 
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user enters handwritten symbols into the system. At step 710, a preprocessor running on 
the MPU operates on the handwritten symbols. At step 720, the RPU operates on the data 
in its memory. At step 730, postprocessor running on the MPU operates on the data to 
generate a symbol observation using the context of the handwritten symbol. 

5 

hi one embodiment, the postprocessor uses spelling information combined with 
previously generated symbols to determine the current symbol. Figure 8 illustrates the 
process of handvmtten symbol translation using spelling information in accordance with 
one embodiment of the present invention. At step 800, a user enters handvmtten symbols 

10 into the system. At step 810, a preprocessor running on the MPU operates on the 

handwritten symbols. At step 820, the RPU operates on the data in its memory. At step 
830, postprocessor running on the MPU adjusts the probability of symbols based on 
whether the symbol, in combination with applicable previous symbols, would produce a 
correctly spelled word or word segment in the current context. At step 840, the 

15 postprocessor generates a symbol observation. For example, the postprocessor would 

assign a higher probability to a symbol following a "q" being a "u" rather than being "ei". 

In another embodiment, the postprocessor uses grammar information combined 
with previously generated symbols to determine the current symbol. Figure 9 illustrates 

20 the process of handwritten symbol translation using grammar information in accordance 
with one embodiment of the present invention. At step 900, a user enters handwritten 
symbols into the system. At step 910, a preprocessor running on the MPU operates on 
the handwritten symbols. At step 920, the RPU operates on the data in its memory. At 
step 930, postprocessor ruiming on the MPU adjusts the probabihty of symbols based on 

25 whether the symbol, in combination with applicable previous symbols, would produce a 

20 
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set of symbols which are not grammatically incorrectly in the current context. At step 
940, the postprocessor generates a symbol observation. For example, a processor would 
give a higher probability to the symbol following "I am an inventor to" being an "o" than 
being a 

5 

In one embodiment, the postprocessor uses past word usage information 
combined with previously generated symbols to determine the current symbol. Figure 10 
illustrates the process of handwritten symbol translation using past word usage 
information in accordance with one embodiment of the present invention. At step 1000, a 
O 10 user enters handwritten symbols into the system. At step 1010, a preprocessor running on 

the MPU operates on the handwritten symbols. At step 1020, the RPU operates on the 
K data in its memory. At step 1030, postprocessor running on the MPU adjusts the 
m probability of symbols based on whether the symbol, in combination with applicable 
= previous symbols, would produce a word, or segment of a word, which was previously 

Ln 15 input to the system. At step 1040, the postprocessor generates a symbol observation. 

In one embodiment, the postprocessor uses user information combined with 
previously generated symbols to determine the current symbol. Figure 1 1 illustrates the 
process of handwritten symbol translation using user information in accordance with one 
20 embodiment of the present invention. At step 1 100, a user enters handwritten symbols 
into the system. At step 1 1 10, a preprocessor running on the MPU operates on the 
handwritten symbols. At step 1 120, the RPU operates on the data in its memory. At step 
1 130, postprocessor running on the MPU adjusts the probability of symbols to account 
for knovm user information. For example, the system may know the user writes "ful" in a 
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consistently distinct manner. At step 1 140, the postprocessor generates a symbol 
observation. 

Thus, a method and apparatus for accelerated handwritten symbol recognition in a 
pen based tablet computer is described in conjunction with one or more specific 
embodiments. The invention is defined by the following claims and their full scope and 
equivalents. 
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